Documentation

Overview

Overview

MetaLab is a research tool for aggregating across studies in the cognitive development literature. Currently, MetaLab contains 2112 effect sizes across meta-analyses in 25 domains of cognitive development, based on data from 547 papers collecting 38105 subjects. These studies can be used to obtain better estimates of effect sizes across different domains, methods, and ages. Using our power calculator, researchers can use these estimates to plan appropriate sample sizes for prospective studies. More generally, MetaLab can be used as a theoretical tool for exploring patterns in development across language acquisition domains. Learn more here.

Documentation

  • For more information about individual datasets, see the Datasets tab.
  • For information about our statistical approach throughout the site, see the Statistical Approach tab.
  • For information about the field structure of the site, see the Field Specification tab.
  • For information about doing meta-analyses and contributing to MetaLab, see the Contribute page.

Caveat

Please note that data and visualizations are under development at the moment (Spring 2018) and should not be taken as definitive.

Datasets

Datasets

This page provides an overview of all datasets in MetaLab at this moment. All datasets have more detailed descriptions available through their home pages.

Cognitive development

Categorization bias

  • Citation:

  • N papers = 9, N effect sizes = 80

  • In a triad-task, bias to generalize to taxonomic as opposed to thematic alternative.

  • Curator: Molly Lewis

  • Comments:

Prosocial agents

  • Citation: Margoni, F., & Surian, L. (2018). Infants’ evaluation of prosocial and antisocial agents: A meta-analysis. Developmental Psychology, 54(8), 1445-1455.

  • N papers = 26, N effect sizes = 61

  • Do infants prefer a prosocial agent (helper, giver, more fair) over an antisocial (hinderer, taker, less fair) agent?

  • Curator: Francesco Margoni

  • Comments:

Simple arithmetic competences

  • Citation: Christodoulou, J., Lac, A., & Moore, D. S. (2017). Babies and math: A meta-analysis of infants’ simple arithmetic competence. Developmental Psychology, 53(8), 1405-1417.

  • N papers = 6, N effect sizes = 14

  • Do infants look longer at stimuli representing “incorrect” versus “correct” solutions of basic addition and subtraction problems?

  • Curator:

  • Comments:

Early language

Abstract rule learning

  • Citation: Rabagliati, H., Ferguson, B., & Lew-Williams, C. (2018). The profile of abstract rule learning in infancy: Meta‐analytic and experimental evidence. Developmental Science.

  • N papers = 20, N effect sizes = 95

  • Curator: Hugh Rabagliati

  • Comments:

Cross-situational word learning

  • Citation: Dal Ben, R., Souza,D. & Hay, J. (2019). Cross-situational word-learning.

  • N papers = 16, N effect sizes = 50

  • Infants and children s abilities to learn words across multiple ambiguous situations.

  • Curator: Rodrigo Dal Ben

  • Comments:

Function word segmentation

  • Citation: Bergmann, C., & Cristia, A. (2016). Development of infants’ segmentation of words from native speech: A meta-analytic approach. Developmental Science, 19, 901–917

  • N papers = 5, N effect sizes = 17

  • Recognition of familiarized function words from natural speech using behavioral methods applied to infants 0 - 20 months of age.

  • Curator: Christina Bergmann

  • Comments:

Familiar word recognition

  • Citation: Carbajal, M.J. (2018). Doctoral dissertation

  • N papers = 16, N effect sizes = 34

  • Within-subject comparisons of attention to lists of familiar words vs novel/rare words.

  • Curator: Julia Carbajal

  • Comments:

Gaze following

  • Citation: Frank, M. C., Lewis, M., & MacDonald, K. (2016). A performance model for early word-learning. In Proceedings of the 38th Annual Conference of the Cognitive Science Society.

  • N papers = 12, N effect sizes = 33

  • Gaze-following using standard multi-alternative forced-choice paradigms, in infants and children from 3 months to two years.

  • Curator: Molly Lewis

  • Comments:

Infant directed speech preference

  • Citation: Dunst, C. J., Gorman, E., & Hamby, D. W. (2012). Preference for infant-directed speech in preverbal young children. Center for Early Literacy Learning, 5(1).

  • N papers = 23, N effect sizes = 75

  • Looking times as a function of whether infant-directed vs. adult-directed speech is presented as stimulation, in infants aged 0 - 12 months.

  • Curator: Alex Cristia

  • Comments:

Label advantage in concept learning

  • Citation:

  • N papers = 18, N effect sizes = 102

  • Previous research suggests that labels support the formation of novel categories in young children. This meta-analysis includes studies exploring this question that use a paradigm similar to Balaban and Waxman (1997), resulting in a selection of experiments using behavioral methods on infants 3 to 18 months.

  • Curator: Molly Lewis

  • Comments:

Mutual exclusivity

  • Citation: Lewis, M., Cristiano, V., Lake, B. M., Kwan, T., & Frank, M. C. (under review). The role of developmental change and linguistic experience in the mutual exclusivity effect.

  • N papers = 48, N effect sizes = 146

  • Mutual exclusivity is the bias for children to assume a novel word refers to a novel object, in the presence of one novel and one familiar object. This meta-analysis covers behavioral experiments on infants aged 14 months to 5 years.

  • Curator: Molly Lewis

  • Comments:

Mispronunciation sensitivity

  • Citation: Von Holzen, K., & Bergmann, C. (2018). A meta-analysis of infants’ mispronunciation sensitivity development. Proceedings of the 40th Annual Conference of the Cognitive Science Society. Austin, TX: Cognitive Science Society, Inc.

  • N papers = 32, N effect sizes = 251

  • Curator: Katie Von Holzen

  • Comments:

Natural speech preference

  • Citation: Issard, C., & Cristia, A. (2019, April 4). Infants prefer to listen to speech: A meta-analysis. Retrieved from osf.io/4stz9.

  • N papers = 18, N effect sizes = 64

  • Do infants prefer natural speech over other species vocalizations, natural or artificial sounds?

  • Curator: Cecile Issard

  • Comments: This meta-analysis is in progress.

Online word recognition

  • Citation: Frank, M. C., Lewis, M., & MacDonald, K. (2016). A performance model for early word-learning. In Proceedings of the 38th Annual Conference of the Cognitive Science Society.

  • N papers = 6, N effect sizes = 15

  • Online word recognition of familiar words using two-alternative forced choice preferential looking. Experiments in this MA follow Fernald et al. (1998) and were included based on their reporting of reaction time as well as effect size.

  • Curator: Molly Lewis

  • Comments:

Phonotactic learning

  • Citation: Cristia, A., (2018). Can infants learn phonology in the lab? A meta-analytic answer. Cognition, 170, 312-327.

  • N papers = 15, N effect sizes = 47

  • Laboratory experiments in which infants are exposed to a set of pseudo-words that could support a phonotactic generalization, and thereafter tested on their preference between new types that follow or violate the familiarized phonotactics.

  • Curator: Alex Cristia

  • Comments:

Pointing and vocabulary (longitudinal)

  • Citation: Colonnesi, C., Stamsa, G. J., Kostera, I., & Noomb, M. J. (2010). The relation between pointing and language development: A meta-analysis. Developmental Review, 30, 352–366.

  • N papers = 18, N effect sizes = 18

  • The pointing gesture has been argued to be related to the development of language. This meta-analysis explores the relationship between pointing and language development, both concurrently and longitudinally.

  • Curator: Molly Lewis

  • Comments:

Pointing and vocabulary (concurrent)

  • Citation: Colonnesi, C., Stamsa, G. J., Kostera, I., & Noomb, M. J. (2010). The relation between pointing and language development: A meta-analysis. Developmental Review, 30, 352–366.

  • N papers = 12, N effect sizes = 12

  • The pointing gesture has been argued to be related to the development of language. This meta-analysis explores the relationship between pointing and language development, both concurrently and longitudinally.

  • Curator: Molly Lewis

  • Comments:

Statistical sound category learning

  • Citation: Cristia, A., (2018). Can infants learn phonology in the lab? A meta-analytic answer. Cognition, 170, 312-327.

  • N papers = 11, N effect sizes = 20

  • Laboratory experiments in which infants are exposed to a set of syllables that have been generated using a bimodal distribution of acoustic cues or not, and infants are tested on their discrimination of two sounds differing along the same dimension.

  • Curator: Alex Cristia

  • Comments:

Statistical word segmentation

  • Citation: Black, A. & Bergmann, C. (2017). Quantifying infants’ statistical word segmentation: A meta-analysis. In: Proceedings of the 39th Annual Conference of the Cognitive Science Society, pp. 124-129, Austin, TX: Cognitive Science Society

  • N papers = 31, N effect sizes = 103

  • Can infants segment words from artificial mini-languages based on syllable-level statistics?

  • Curator: Christina Bergmann

  • Comments:

Sound symbolism

  • Citation: Fort, M., Lammertink, I., Peperkamp, S., Guevara-Rukoz, A., Fikkert, P., & Tsuji, S. (2018). SymBouki: a meta-analysis on the emergence of sound symbolism in early language acquisition. Developmental Science, Early View. doi: 10.1111/desc.12659.

  • N papers = 11, N effect sizes = 44

  • Do infants recognize systematic relationships between form and meaning, e.g. that bouba tends to refer to a round object and kiki tends to refer to a spiky one?

  • Curator: Sho Tsuji

  • Comments:

Switch Task

  • Citation: Tsui, A. S. M., Byers-Heinlein, K., & Fennell, C. T. (2019, February 7). Associative Word Learning in Infancy: A Meta-Analysis of the Switch Task. Developmental Psychology. Advance online publication.

  • N papers = 47, N effect sizes = 143

  • Curator: Angeline Tsui

  • Comments:

Symbolic play

  • Citation: Quinn, S., Donnelly, S., & Kidd, E. (2018). The relationship between symbolic play and language acquisition: A meta-analytic review. Developmental review, 49, 121-135.

  • N papers = 31, N effect sizes = 196

  • Do infants who show more symbolic play have better language development?

  • Curator:

  • Comments: the mean age is not systematic, but the center of the age range provided in the supplementary material.

Vowel discrimination (native)

  • Citation: Tsuji, S. & Cristia, A. (2014). Perceptual attunement in vowels: A meta-analysis. Developmental Psychobiology, 56(2), 179-191.

  • N papers = 33, N effect sizes = 145

  • Discrimination of native-language vowels, including results from behavioral, electrophysiological, and neuroimaging methods applied to infants 0 - 14 months of age.

  • Curator: Sho Tsuji

  • Comments:

Vowel discrimination (non-native)

  • Citation: Tsuji, S. & Cristia, A. (2014). Perceptual attunement in vowels: A meta-analysis. Developmental Psychobiology, 56(2), 179-191.

  • N papers = 15, N effect sizes = 49

  • Discrimination of non-native language vowels, including results from behavioral, electrophysiological, and neuroimaging methods applied to infants 0 - 14 months of age.

  • Curator: Sho Tsuji

  • Comments:

Word segmentation

  • Citation: Bergmann, C., & Cristia, A. (2016). Development of infants’ segmentation of words from native speech: A meta-analytic approach. Developmental Science, 19, 901–917

  • N papers = 68, N effect sizes = 298

  • Recognition of familiarized content or non-words from running, natural speech using behavioral methods applied to infants 0 - 20 months of age.

  • Curator: Christina Bergmann

  • Comments:

Contributing to MetaLab

How to contribute?

We welcome researchers interested in contributing to Metalab. Please contact us.

Contributions can take various forms:

  • Suggesting and/or adding published papers not yet included in a meta-analysis. If you see a study we missed, just email the curator, ideally including the pdf. If you are the author, try to make sure that all necessary data is either included or provided in your email.
  • Adding unpublished data to an existing meta-analysis. Please email the curator of the meta-anlysis you wish to complete and we will tell you how we can go about adding your unpublished data to our meta-analyses.
  • Creating new meta-analyses. If you are interested in creating a new meta-analysis, check out the tutorial page.
  • Contribute existing meta-analyses not yet in MetaLab. If you already have a completed meta-analysis, make sure to use the MetaLab template and get in touch about integrating it into MetaLab. Regarding the ownership of your data, you remain the owner of your meta-analysis data, and researchers publishing or presenting work using data in MetaLab must cite the publications linked to your meta-analysis. We currently offer two options for the meta-analysis maintenance and update:
  • Be the curator. This means the following:
    • You are responsible for identifying new relevant papers (although the MetaLab data manager can add them for you). We recommend to set alerts for new papers on your meta-analysis topic.
    • You should check data entry and correct it if needed once in a while.
    • You determine moderators to be displayed and are responsible of executive decisions regarding spreadsheet content.
    • By default you will be the curator for one year. Each year we will send an email listing all the meta-analysis you are in charge of and asking you if you still want to be the curator of them. Curators are part of the MetaLab board and get informed of discussions regarding e.g. site revamping.
  • Step down completely, and it will be MetaLab’s job to assign a new curator for your dataset. In this case, we can still keep your photo on the wall of fame.

Check out the tutorial page on how to edit your meta-analysis in the MetaLab format.

Statistical Approach

Overview

All analyses on the site are conducted with the metafor package (Viechtbauer, 2010).

Effect Size Computation

Effect size computation is handled by a script, compute_es.R.

Several pre-existing MAs deal with special cases, and these are listed in the script.

Except where noted, formulas are from Hedges & Olkin’s textbook.

Statistical Models

The visualizations page uses a multi-level random effects meta-analysis ( rma.mv function of metafor). Random-effect models assume that the true effect can vary between different studies, and therefore allow random effects for each data point. The model also specifies random effects on the level of each paper, since studies within a paper can be assumed to be more similar to each other than to studies from different papers. In addition, we allow correlated random effects within each paper to account for cases where the same infants contributed data to multiple rows.

The meta-analytic models are accessible in the script server.R

Field Specifications

MetaLab promotes open science practices. All the datasets of the meta-analyses appearing on the website can be downloaded. If you want to work with one of them, these instructions might be useful:

Field Specifications

This page gives the full specification for each field in the metalab dataset, including: required fields (which must be included for every MA), optional fields (which are only used for some MAs), and derived fields (which are computed by the site).

Required fields
Optional fields
Derived fields

Export Data

Download the data

You can download the data from both the home page and the visualization page.

  1. From the home page

    1.1. Click on the box corresponding to the domain you are interested in (e.g. cognitive development, early language).

    1.2. Once you are on the page of the domain, click on the box corresponding to the meta-analysis you are interested in (e.g. word segmentation).

    1.3. In the data tab, click on the “Download” button.

    1.4. Choose the data format that you want (EXCEL - recommended for manual calculations using a spreadsheet software - but you should make sure that your local spreadsheet software is set to use “.” as decimal separator to avoid errors (see section 2.), or CSV - recommended for reading into a statistical software such as R) by clicking on it in the conditional menu.

  2. From the visualization page.

    2.1. Open the “Domain” drop-down menu and click on the one you are interested in.

    2.2. Open the “Dataset” drop-down menu and click on the meta-analysis you are interested in.

    2.3. Click on the “Download data” button above the “Domain” menu. You can also click on “View raw dataset” and follow steps 1.2. and 1.3.

Open the data

The following instructions assume you are using Excel:

  1. Open a blank workbook

  2. Go to the data menu

  3. Click on “Get external data”

  4. Click on “From text”

  5. A dialog box opens. In the drop-down menu on the right, select “UTF-8”.

  6. Select “Delimited” and click “next”

  7. Select “comma” as column delimiter and click “next”

  8. Click on the the “advanced” button and check that “.” is the decimal separator.

  9. Click OK to finish.

Working on a subset of data.

Your research question may be more specific than the one of the meta-analysis. For example your question may focus on a more specific age than the meta-analysis, or you are using a specific method.

  1. Click on “Data” > “Filter” (libre/openoffice have similar menus)

  2. Scroll to the column coding your criteria (e.g. “mean_age_1”)

  3. Click on the little triangle/filter symbol on the right corner of that column

  4. Click on “Standard filter” or “Special filter” or “Custom filter”

  5. In the next dialogue box, choose your condition to be that the column entitled “mean_age_1” is lower than the maximum age (in days) your participants-to-be will have, and greater than your lower age bound.* (We use the conversion: month x 30.42 days/month). If you want to add a second condition, don’t press OK just yet!

  6. To add a second condition: that the “method” is your chosen method (see here for codes). You SHOULD keep only studies with your chosen method, but other variables are left to your judgement. For instance, if I were doing a study on 6 mo using Central Fixation, then my filters could be: method=“CF” AND 120 < mean_age_1 <=365 (so babies between about 4 months and 1 year).

  7. You can add as many conditions as you want (e.g., exclude some more studies because they use unusual stimuli, or a very different design from yours).

  8. When you’re done entering inclusion conditions, you can click OK. The result will be a set of rows that contain all the effect sizes meeting your conditions.

  • Note: If you do not see options for greater/lower but rather lists of values, that means the import didn’t work correctly and these numeric values are seen as text. See the “Open the data” - troubleshooting section.

Sample size estimation

Two possibilities:

  1. Compute the mean or the median of column “n_1”, and “n_1” and “n_2” if you are running a between participants study. You can thus see how many infants per group you should test based on the sample sizes used in previous studies. We do not recommend this strategy.

    1.1. In a new cell (typically below the last row of the “n_1” and “n_2” columns), type “=Mean([coordinate (i.e. letter+number) of the first value]:[coordinate (i.e. letter+number) of the last value])” or “=median([coordinate (i.e. letter+number) of the first value]:[coordinate (i.e. letter+number) of the last value])”. You can also type “=mean(“ and select the the cells to average with the cursor. Make sure you’ve closed the bracket and press enter.

    1.2. Note that if the variable “n_excluded_1” is coded in the meta-analysis you are looking at, you can also get an idea of the attrition rates found in this work. For instance, you can add a column, say called “n_total”, that sums n_1 and n_excluded_1. This will give you an idea of how many babies you should recruit, taking into account that some of them will need to be excluded (because they are fussy or other reasons).

    1.3. If n_excluded_1 is not present, you can use a 20% rule of thumb - i.e., recruit 20% more than your target sample size. You can get an idea of the attrition rate by method and age group on our design choices analyses page.

  2. Previous studies might have chosen their sample size based on practical, instead of statistical reasons -e.g. number of available participants within a predefined time range. Therefore they might be underpowered. To solve the problem on your study, you can do a prospective power analysis.

    2.1. Copy the column entitled d_calc into a new sheet.

    2.2. Calculate the median values for these columns – in my example, the median difference score is 0.91, the median pooled SD is 2.72, and the median effect size is 0.33.

    2.3. Use your favorite power calculation system/online tool to estimate sample size. I used the following formula in R (package: pwr): power.t.test(delta=0.91,sd=2.72,power=0.8,sig=.05,type=c(“paired”),alternative=“two.sided”) to find out I needed to test 72 infants for my example case.

If the number you get is too high, you might consider running sequential analyses. This means that you fully design your study, including a predetermined number of participants to run, the statistical analyses that you will run, and predefine moments when you will look at p-values before the end of data collection. At these predefined moments you will decide to continue or stop data collection based on outcome (significance reached, or fell below the threshold).

You can also define a stopping rule related to your data collection situation - e.g. “Data collection will stop when all conditions are balanced (N=72) or by Dec 31, 2018 (date when my internship ends), whichever happens first”. Notice that neither of these options takes power into consideration - they are not alternatives to power, they just help you avoid the questionable practice of stopping data collection when you have p<.05.

Citations

You must cite the publications linked to that dataset, as listed in the Documentation. (If there is no citation, then no citation for the individual dataset is necessary).

For tracking use of MetaLab, please also cite at least one of the following:

Bergmann, C., Tsuji, S., Piccinini, P.E., Lewis, M.L., Braginsky, M., Frank, M.C., & Cristia, A. (2018). Promoting replicability in developmental research through meta-analyses: Insights from language acquisition research. Child Development, 89, 1996-2009 . DOI: 10.1111/cdev.13079(https://osf.io/uhv3d/)[Repository]

Lewis, M. L., Braginsky, M., Tsuji, S., Bergmann, C., Piccinini, P. E., Cristia, A., & Frank, M. C. (2017/under review). A Quantitative Synthesis of Early Language Acquisition Using Meta-Analysis. DOI: 10.17605/OSF.IO/HTSJM (https://psyarxiv.com/htsjm)[Preprint]

If researchers use more than five datasets, we ask users as a courtesy to cite all data. But in case of severe space limitations, the database as a whole alone may be cited.

Since the MetaLab site is dynamic, we recommend that you list the date of download for your data in your manuscript. For example: “We analyze all data currently in MetaLab (Bergmann et al., 2018). Data were downloaded on 06/17/17.”

Troubleshooting

We are collecting frequent problems here. If you experience any issues that we do not cover when following the steps, please send an email to [metalab-project@googlegroups.com] and we will try to help you.

Import issues:

I see weird symbols.

Make sure you selected UTF-8 as the encoding when you imported the data (see steps 2.1 to 2.5).

Some of the rows have more columns than others

This is due to a faulty import, you need to do it again:

  1. Open a blank workbook

  2. Go to the data menu

  3. Click on “Get external data”

  4. Click on “From text”

  5. A dialog box opens. In the drop-down menu on the right, select “UTF-8”.

  6. Select “Delimited” and click “next”

  7. Select “comma” as column delimiter and click “next”

  8. Click on the the “advanced” button and check that “.” is the decimal separator.

  9. Click OK to finish.

Numbers are formatted weirdly, or appear as text.

Check that . is the decimal separator:

  1. Select the column where numbers are not formatted properly (if there are several, you need to treat them one by one).

  2. Click on “data” > “Text to columns”. A dialog box opens

  3. Select “Delimited” and click “next”

  4. Select “comma” as column delimiter and click “next”

  5. Click on the the “advanced” button and check that “.” is the decimal separator. Click OK to finish.

When I open in Excel, all the data are in column A.
  1. select column A (that contains all the data)

  2. Click on “data” > “Text to columns”. A dialog box opens

  3. Select “Delimited” and click “next”

  4. Select “comma” as column delimiter and click “next”

  5. Click on the the “advanced” button and check that “.” is the decimal separator. Click OK to finish.